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1. Introduction 


Logical abilities are a ubiquitous ingredient in all those contexts that take into account soft 
skills, argumentative skills, or critical thinking. However, there is a substantial lack of research 
that addresses the actual possession of such logical abilities by students. With this aim, since 
October 2020 the University of Florence has promoted a three-stage initiative to collect data in 
order to measure the logical abilities of students when enrolling at the University. The first stage 
is an entrance test for assessing the students’ initial abilities. This test comprises ten questions, 
each investigating a specific reasoning construct. 

At the second stage, students attend a short training course to strengthen their logical abili- 
ties. As third step, in order to evaluate the effectiveness of the course, they take an exit exami- 
nation, replicating the structure and the difficulty of the entrance test. 

This paper builds on the previous work by Bertaccini et al. (2021) where the effectiveness 
of the course was tested via Item Response Theory (DeMars, 2010; Bartolucci et al., 2019) 
and test-equating techniques (Battauz, 2015). Building on an enlarged database of students that 
took the training course and examinations in the second half of 2021 and leveraging auxiliary 
information about students’ characteristics, we estimated a Structural Equation Model (SEM; 
Duncan, 2014; Bollen, 1989) to have a better comprehension and interpretation of the results 
reported by Bertaccini et al. (2021). 


2. Data and methods 


Data 


The data that we analyse in this work are obtained from the 80 students that took both the tests 
and the short training course in the second semester of the academic year 2021-2021. The items 
of each test aimed at investigating the same logical constructs, namely: Double negation (item 
code N); Disjunction negation (item code D); Conjunction negation (item code C); Hypothetical 
reasoning (item code IMPL); Sufficient and necessary conditions (item code NEC); Negation 
of the universal quantifier (item code NU); Negation of the existential quantifier (item code 
NE); Modus tollens (item code MT); Syllogism (item code S); Multiple steps deduction (item 
code DED). The students who respond correctly to a given item are given a score equal to 1, 
otherwise they are given a score equal to 0. 

In addition, we were able to obtain exogenous information on students’ characteristics such 
as their age, the grade obtained at the secondary school, the scientific area the student has 
enrolled in (i.e. science, social, technic, humanistic) and the years of university enrolment. 

Compared to the previous work by Bertaccini et al. (2021), the novelty of this study consists 
in investigating the role of auxiliary information on students to explain their logic abilities and 
the effectiveness of the training course. The authors assessed the effectiveness of the course by 
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first estimating two Item Response Theory (IRT) models, one for the entrance test and one for 
the exit test, and then it tested whether there was a significant shift in the distribution of the log- 
ical abilities through a test equating procedure. Given the availability of auxiliary information, 
we opted for a one-step procedure based on SEM so as to take into account both the measure- 
ment issue of logical abilities before and after the training course and the structural relations 
among the observed (i.e., student characteristics) and the latent variables (i.e., logical abilities). 


Methods 


A SEM is a multivariate technique used to test complex relationships between observed (man- 
ifest) and unobserved (latent) variables as well as relationships between two or more latent 
variables. Special observed variables, named indicators or items, are used to measure the latent 
variables. In turn, observed and latent variables distinguish in exogenous variables, which are 
not explained within the model, and endogenous variables that are affected by other variables 
in the model (plus an error term). A SEM is characterised by a system of multiple equations, 
discerning between two sub-models: (i) a structural model, designed to explain the relationships 
among latent variables as well as among endogenous latent variables and observed variables, 
and (ii) a measurement model, to link the latent variables to the items. In more detail, the 
structural model can be expressed by the following equation 


n= Bn+Té+¢, (1) 


where we model the latent logic ability at the exit test 7 as depending on the latent logic 
ability at entrance, €. Also, in (1) B is a matrix of regression coefficients of the endogenous 
latent variables; I is the matrix of regression coefficients among the endogenous and exogenous 
latent variables and ¢ is vector of errors. 


Figure 1: Structural part of the theoretical SEM. 


The measurement model is defined by two equations, respectively for the endogenous (2) 
and exogenous (3) latent variables: 


a=A,C+o, (2) 


38 


y=Ayn +e, (3) 


where y is a vector of the item responses, æ is a vector of exogenous individual characteris- 
tics. In both (2) and (3) A, and A, are matrices of factor loading while 6 and ø are vectors of 
error terms. 

Note that when one or more exogenous variables are not affected by measurement errors, 
the structural (1) is simplified as: 


n= Bn+Ta+¢ (4) 


3. Results 


The proposed SEM with all the significant variables is reported in Figure 1. More detailed 
estimates are shown in Table 1. The estimated SEM presents a good fit to the data with Com- 
parative Fit Index (CFI) equal to 0.944, Tucker-Lewis Index (TLI) of 0.935 and a Root Mean 
Squared Error of Approximation (RMSEA) of 0.054. All estimates were obtained using the 
R-package Lavaan (Rosseel, 2012). 


Figure 2: Final SEM. 


Our results for the measurement part suggest that the course was indeed effective as the 
estimated items’ coefficients are greater in magnitude after having attended the training course. 
(see Figure 1). 

Regarding the regression part of the model, we found that the only significant determinant of 
the logical skill at entrance was the final grade obtained at the high-school. Also, we found that 
the only significant effect on the logical skill at exit was the logical skill at entrance. These re- 
sults confirms that the short training course was indeed useful to sharp students’ logical abilities 
and moreover it is consistent with the preliminary results obtained by Bertaccini et al. (2021). 
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Table 1: SEM results: measurement part, regression part, and covariances. 
Estimate Std.Err z-value P(>|z|) Stdlv Std.all 


Measurement: 

Dep: Ability IN 

N1 1.000 0.650 0.633 
D1 0.755 0.257 2.936 0.003 0.490 0.483 
C1 0.893 0.317 2.814 0.005 0.580 0.568 
Sl 0.407 0.277 1.471 0.141 0.265 0.264 
IMPLI 0.839 0.495 1.696 0.090 0.545 0.535 
NEC1 0.831 0.357 2.329 0.020 0.540 0.530 
NUI 0.909 0.268 3.392 0.001 0.591 0.578 
NEI 1.084 0.325 3.339 0.001 0.704 0.683 
MT1 0.664 0.284 2.341 0.019 0.431 0.426 
DED1 0.544 0.260 2.097 0.036 0.354 0.351 
Dep: Ability OUT 

N2 1.000 0.298 0.298 
D2 2.477 1.367 1.812 0.070 0.739 0.728 
C2 2.234 1.275 1.752 0.080 0.666 0.659 
S2 1.548 1.151 1.345 0.179 0.462 0.459 
IMPL2 2.467 1.358 1.816 0.069 0.736 0.726 
NEC2 1.755 1.002 1.751 0.080 0.523 0.520 
NU2 1.791 1.253 1.429 0.153 0.534 0.530 
NE2 3.384 1.925 1.758 0.079 1.009 0.983 
MT2 1.315 0.918 1.433 0.152 0.392 0.391 
DED2 2.104 1.167 1.802 0.071 0.627 0.621 
Regression: 

Dep: Ability IN 

votomat 0.754 0.317 2.375 0.018 1.160 0.359 
Dep: Ability OUT 

Ability IN 0.525 0.232 2.263 0.024 0.639 0.639 
Covariances: 

NEC1;NEC2 0.498 0.143 3.480 0.001 0.498 0.671 


C1;C2 0.412 0.125 3.296 0.001 0.412 0.644 
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4. Conclusions 


In this paper, we took extended the previous work study by Bertaccini et al. (2021) to offer 
a more comprehensive and a unified framework to test the effectiveness of the training course 
for the development of the logical skills of students enrolling at the University of Florence. The 
effectiveness of the course is confirmed thus making advisable for the University of Florence to 
design an internal policy so that it may become a standard tool of training and evaluation. 
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